A robust and parsimonious machine learning method to predict ICU admission of COVID-19 patients.

In this article, we discuss the development of prognostic machine learning (ML) models for COVID-19 progression, by focusing on the task of predicting ICU admission within (any of) the next 5 days. On the basis of 6,625 complete blood count (CBC) tests from 1,004 patients, of which 18% were admitted to intensive care unit (ICU), we created four ML models, by adopting a robust development procedure which was designed to minimize risks of bias and over-fitting, according to reference guidelines. The best model, a support vector machine, had an AUC of .85, a Brier score of .14, and a standardized net benefit of .69: these scores indicate that the model performed well over a variety of prediction criteria. We also conducted an interpretability study to back up our findings, showing that the data on which the developed model is based is consistent with the current medical literature. This also demonstrates that CBC data and ML methods can be used to predict COVID-19 patients' ICU admission at a relatively low cost: in particular, since CBC data can be quickly obtained by means of routine blood exams, our models could be used in resource-constrained settings and provide health practitioners with rapid and reliable indications.

A Confidence Interval-Based Method for Classifier Re-Calibration.

We propose a re-calibration method for Machine Learning models, based on computing confidence intervals for the predicted confidence scores. We show the effectiveness of the proposed method on a COVID-19 diagnosis benchmark.

How is test laboratory data used and characterised by machine learning models? A systematic review of diagnostic and prognostic models developed for COVID-19 patients using only laboratory data.

The current gold standard for COVID-19 diagnosis, the rRT-PCR test, is hampered by long turnaround times, probable reagent shortages, high false-negative rates and high prices. As a result, machine learning (ML) methods have recently piqued interest, particularly when applied to digital imagery (X-rays and CT scans). In this review, the literature on ML-based diagnostic and prognostic studies grounded on hematochemical parameters has been considered. By doing so, a gap in the current literature was addressed concerning the application of machine learning to laboratory medicine. Sixty-eight articles have been included that were extracted from the Scopus and PubMed indexes. These studies were marked by a great deal of heterogeneity in terms of the examined laboratory test and clinical parameters, sample size, reference populations, ML algorithms, and validation approaches. The majority of research was found to be hampered by reporting and replicability issues: only four of the surveyed studies provided complete information on analytic procedures (units of measure, analyzing equipment), while 29 provided no information at all. Only 16 studies included independent external validation. In light of these findings, we discuss the importance of closer collaboration between data scientists and medical laboratory professionals in order to correctly characterise the relevant population, select the most appropriate statistical and analytical methods, ensure reproducibility, enable the proper interpretation of the results, and gain actual utility by using machine learning methods in clinical practice.

Machine Learning for Health: Algorithm Auditing & Quality Control.

Developers proposing new machine learning for health (ML4H) tools often pledge to match or even surpass the performance of existing tools, yet the reality is usually more complicated. Reliable deployment of ML4H to the real world is challenging as examples from diabetic retinopathy or Covid-19 screening show. We envision an integrated framework of algorithm auditing and quality control that provides a path towards the effective and reliable application of ML systems in healthcare. In this editorial, we give a summary of ongoing work towards that vision and announce a call for participation to the special issue  Machine Learning for Health: Algorithm Auditing & Quality Control in this journal to advance the practice of ML4H auditing.

External validation of Machine Learning models for COVID-19 detection based on Complete Blood Count.

PURPOSE: The rRT-PCR for COVID-19 diagnosis is affected by long turnaround time, potential shortage of reagents, high false-negative rates and high costs. Routine hematochemical tests are a faster and less expensive alternative for diagnosis. Thus, Machine Learning (ML) has been applied to hematological parameters to develop diagnostic tools and help clinicians in promptly managing positive patients. However, few ML models have been externally validated, making their real-world applicability unclear. METHODS: We externally validate 6 state-of-the-art diagnostic ML models, based on Complete Blood Count (CBC) and trained on a dataset encompassing 816 COVID-19 positive cases. The external validation was performed based on two datasets, collected at two different hospitals in northern Italy and encompassing 163 and 104 COVID-19 positive cases, in terms of both error rate and calibration. RESULTS AND CONCLUSION: We report an average AUC of 95% and average Brier score of 0.11, out-performing existing ML methods, and showing good cross-site transportability. The best performing model (SVM) reported an average AUC of 97.5% (Sensitivity: 87.5%, Specificity: 94%), comparable with the performance of RT-PCR, and was also the best calibrated. The validated models can be useful in the early identification of potential COVID-19 patients, due to the rapid availability of CBC exams, and in multiple test settings.

The importance of being external. methodological insights for the external validation of machine learning models in medicine.

Background and Objective Medical machine learning (ML) models tend to perform better on data from the same cohort than on new data, often due to overfitting, or co-variate shifts. For these reasons, external validation (EV) is a necessary practice in the evaluation of medical ML. However, there is still a gap in the literature on how to interpret EV results and hence assess the robustness of ML models. METHODS: We fill this gap by proposing a meta-validation method, to assess the soundness of EV procedures. In doing so, we complement the usual way to assess EV by considering both dataset cardinality, and the similarity of the EV dataset with respect to the training set. We then investigate how the notions of cardinality and similarity can be used to inform on the reliability of a validation procedure, by integrating them into two summative data visualizations. RESULTS: We illustrate our methodology by applying it to the validation of a state-of-the-art COVID-19 diagnostic model on 8 EV sets, collected across 3 different continents. The model performance was moderately impacted by data similarity (Pearson ρ = 0.38, p< 0.001). In the EV, the validated model reported good AUC (average: 0.84), acceptable calibration (average: 0.17) and utility (average: 0.50). The validation datasets were adequate in terms of dataset cardinality and similarity, thus suggesting the soundness of the results. We also provide a qualitative guideline to evaluate the reliability of validation procedures, and we discuss the importance of proper external validation in light of the obtained results. CONCLUSIONS: In this paper, we propose a novel, lean methodology to: 1) study how the similarity between training and validation sets impacts the generalizability of a ML model; 2) assess the soundness of EV evaluations along three complementary performance dimensions: discrimination, utility and calibration; 3) draw conclusions on the robustness of the model under validation. We applied this methodology to a state-of-the-art model for the diagnosis of COVID-19 from routine blood tests, and showed how to interpret the results in light of the presented framework.

A Proposal For COVID-19 Applications Enabling Extensive Epidemiological Studies.

During the next phase of COVID-19 outbreak, mobile applications could be the most used and proposed technical solution for monitoring and tracking, by acquiring data from subgroups of the population. A possible problem could be data fragmentation, which could lead to three harmful effects: i) data could not cover the minimum percentage of the people for monitoring efficacy, ii) it could be heavily biased due to different data collection policies, and iii) the app could not monitor subjects moving across different zones or countries. A common approach could solve these problems, defining requirements for the selection of observed data and technical specifications for the complete interoperability between different solutions. This work aims to integrate the international framework of requirements in order to mitigate the known issues and to suggest a method for clinical data collection that ensures to researchers and public health institution significant and reliable data. First, we propose to identify which data is relevant for COVID-19 monitoring through literature and guidelines review. Then we analysed how the currently available guidelines for COVID-19 monitoring applications drafted by European Union and World Health Organization face the issues listed before. Eventually we proposed the first draft of integration of current guidelines.

Evidence of significant difference in key COVID-19 biomarkers during the Italian lockdown strategy. A retrospective study on patients admitted to a hospital emergency department in Northern Italy.

BACKGROUND: The Lombardy region, Italy, has been severely affected by COVID-19. During the epidemic peak, in March 2020, patients needing intensive care unit treatments were approximately 10% of those infected. This fraction decreased to approximately 2% in the second part of April, and to 0.4% at the beginning of July. COVID-19 is characterized by several biochemical abnormalities whose discrepancy from normal values was associated to the severity of the disease. The aim of this retrospective study was to compare the biochemical patterns of patients during and after the pandemic peak in order to verify whether later patients were experiencing a milder COVID-19 course, as anecdotally observed by several clinicians of the same Hospital. MATERIAL AND METHODS: The laboratory findings of two equivalent groups of 84 patients each, admitted at the emergency department of the San Raffaele Hospital (Milan, Italy), during March and April respectively, were analyzed and compared.  Results. White blood cell, platelets, lymphocytes and lactate dehydrogenase showed a statistically significant improvement (i.e. closer or within the normal clinical range) in the April group compared to March. Creatinine, C-reactive protein, Calcium and liver enzymes, were also pointing in that direction, although the differences were not significant. DISCUSSION: The laboratory findings analyzed in this study were consistent with a milder COVID-19 course in the April group. After excluding several hypotheses, we concluded that our observation was likely the consequence of the lockdown strategy enforcement, which, by imposing social distancing and the use of respiratory protective devices, reduced viral loads upon infection.

Development, evaluation, and validation of machine learning models for COVID-19 detection based on routine blood tests.

Objectives: The rRT-PCR test, the current gold standard for the detection of coronavirus disease (COVID-19), presents with known shortcomings, such as long turnaround time, potential shortage of reagents, false-negative rates around 15-20%, and expensive equipment. The hematochemical values of routine blood exams could represent a faster and less expensive alternative. Methods: Three different training data set of hematochemical values from 1,624 patients (52% COVID-19 positive), admitted at San Raphael Hospital (OSR) from February to May 2020, were used for developing machine learning (ML) models: the complete OSR dataset (72 features: complete blood count (CBC), biochemical, coagulation, hemogasanalysis and CO-Oxymetry values, age, sex and specific symptoms at triage) and two sub-datasets (COVID-specific and CBC dataset, 32 and 21 features respectively). 58 cases (50% COVID-19 positive) from another hospital, and 54 negative patients collected in 2018 at OSR, were used for internal-external and external validation. Results: We developed five ML models: for the complete OSR dataset, the area under the receiver operating characteristic curve (AUC) for the algorithms ranged from 0.83 to 0.90; for the COVID-specific dataset from 0.83 to 0.87; and for the CBC dataset from 0.74 to 0.86. The validations also achieved good results: respectively, AUC from 0.75 to 0.78; and specificity from 0.92 to 0.96. Conclusions: ML can be applied to blood tests as both an adjunct and alternative method to rRT-PCR for the fast and cost-effective identification of COVID-19-positive patients. This is especially useful in developing countries, or in countries facing an increase in contagions.

Role of time-normalized laboratory findings in predicting COVID-19 outcome.

Objectives The pandemic COVID-19 currently reached 213 countries worldwide with nearly 9 million infected people and more than 460,000 deaths. Although several Chinese studies, describing the laboratory findings characteristics of this illness have been reported, European data are still scarce. Furthermore, previous studies often analyzed the averaged laboratory findings collected during the entire hospitalization period, whereas monitoring their time-dependent variations should give more reliable prognostic information. Methods We analyzed the time-dependent variations of 14 laboratory parameters in two groups of COVID-19 patients with, respectively, a positive (40 patients) or a poor (42 patients) outcome, admitted to the San Raffaele Hospital (Milan, Italy). We focused mainly on laboratory parameters that are routinely tested, thus, prognostic information would be readily available even in low-resource settings. Results Statistically significant differences between the two groups were observed for most of the laboratory findings analyzed. We showed that some parameters can be considered as early prognostic indicators whereas others exhibit statistically significant differences only at a later stage of the disease. Among them, earliest indicators were: platelets, lymphocytes, lactate dehydrogenase, creatinine, alanine aminotransferase, C-reactive protein, white blood cells and neutrophils. Conclusions This longitudinal study represents, to the best of our knowledge, the first study describing the laboratory characteristics of Italian COVID-19 patients on a normalized time-scale. The time-dependent prognostic value of the laboratory parameters analyzed in this study can be used by clinicians for the effective treatment of the patients and for the proper management of intensive care beds, which becomes a critical issue during the pandemic peaks.

Routine blood tests as an active surveillance to monitor COVID-19 prevalence. A retrospective study.

BACKGROUND: In Italy, one of the country most affected by the COVID-19 pandemic, the first autochthonous case appeared in Lombardy on February 20th, 2020. One month later, the number of -COVID-19 patients in Lombardy exceeded 17000 and about 3500 had died. Because of this rapid increase in infected people scientists wonder whether SARS-CoV-2 was already highly circulating in Lombardy before such date. Plasma levels of aspartate aminotransferase (AST) and lactate dehydrogenase (LDH) were shown to be -highly increased in COVID-19 patients. Monitoring their levels in Emergency Room patients during the months preceding February 20th, 2020, might shade light on the prevalence of the disease in the pre-COVID-19 period. METHODS: We retrospectively analyzed the AST and LDH levels from more than 30.000 patients admitted to the San Raffaele Hospital Emergency Room (ER) between September 2019 and May 2020 as well as between September 2018 and May 2019. The number of patients diagnosed with respiratory tract diseases were also analyzed. RESULTS: Data showed that the ER averaged AST and LDH levels are highly sensitive to the presence of COVID-19 patients. During, the months preceding February 20th, 2020, AST and LDH levels, as well as the number of patients diagnosed with respiratory tract diseases were similar to their 2019 counterparts. CONCLUSIONS: No significant evidence showing that a large number of COVID-19 patients were admitted to the San Raffaele Hospital ER before February 20th, 2020, was found. Thus, the virus was likely circulating, within the Hospital catchment area, either in low amounts or through asymptomatic individuals. Because of the high LDH and AST levels' variations induced by COVID-19, routine blood tests might be exploited as a surveillance indicator for a possible second wave.

Detection of COVID-19 Infection from Routine Blood Exams with Machine Learning: A Feasibility Study.

The COVID-19 pandemia due to the SARS-CoV-2 coronavirus, in its first 4 months since its outbreak, has to date reached more than 200 countries worldwide with more than 2 million confirmed cases (probably a much higher number of infected), and almost 200,000 deaths. Amplification of viral RNA by (real time) reverse transcription polymerase chain reaction (rRT-PCR) is the current gold standard test for confirmation of infection, although it presents known shortcomings: long turnaround times (3-4 hours to generate results), potential shortage of reagents, false-negative rates as large as 15-20%, the need for certified laboratories, expensive equipment and trained personnel. Thus there is a need for alternative, faster, less expensive and more accessible tests. We developed two machine learning classification models using hematochemical values from routine blood exams (namely: white blood cells counts, and the platelets, CRP, AST, ALT, GGT, ALP, LDH plasma levels) drawn from 279 patients who, after being admitted to the San Raffaele Hospital (Milan, Italy) emergency-room with COVID-19 symptoms, were screened with the rRT-PCR test performed on respiratory tract specimens. Of these patients, 177 resulted positive, whereas 102 received a negative response. We have developed two machine learning models, to discriminate between patients who are either positive or negative to the SARS-CoV-2: their accuracy ranges between 82% and 86%, and sensitivity between 92% e 95%, so comparably well with respect to the gold standard. We also developed an interpretable Decision Tree model as a simple decision aid for clinician interpreting blood tests (even off-line) for COVID-19 suspect cases. This study demonstrated the feasibility and clinical soundness of using blood tests analysis and machine learning as an alternative to rRT-PCR for identifying COVID-19 positive patients. This is especially useful in those countries, like developing ones, suffering from shortages of rRT-PCR reagents and specialized laboratories. We made available a Web-based tool for clinical reference and evaluation (This tool is available at https://covid19-blood-ml.herokuapp.com/ ).